An Open Source and Open Hardware Deep Learning-Powered Visual Navigation Engine for Autonomous Nano-UAVs

Nano-size unmanned aerial vehicles (UAVs), with few centimeters of diameter and sub-10 Watts of total power budget, have so far been considered incapable of running sophisticated visual-based autonomous navigation software without external aid from base-stations, ad-hoc local positioning infrastructure, and powerful external computation servers. In this work, we present what is, to the best of our knowledge, the first 27g nano-UAV system able to run aboard an end-to-end, closed-loop visual pipeline for autonomous navigation based on a state-of-the-art deep-learning algorithm, built upon the open-source CrazyFlie 2.0 nano-quadrotor. Our visual navigation engine is enabled by the combination of an ultra-low power computing device (the GAP8 system-on-chip) with a novel methodology for the deployment of deep convolutional neural networks (CNNs). We enable onboard real-time execution of a state-of-the-art deep CNN at up to 18Hz. Field experiments demonstrate that the system's high responsiveness prevents collisions with unexpected dynamic obstacles up to a flight speed of 1.5m/s. In addition, we also demonstrate the capability of our visual navigation engine of fully autonomous indoor navigation on a 113m previously unseen path. To share our key findings with the embedded and robotics communities and foster further developments in autonomous nano-UAVs, we publicly release all our code, datasets, and trained networks

Fly, Wake-up, Find: UAV-based Energy-efficient Localization for Distributed Sensor Nodes

A challenging application scenario in the field of industrial Unmanned Aerial Vehicles (UAVs) is the capability of a robot to find and query smart sensor nodes deployed at arbitrary locations in the mission area. This work explores the combination of different communication technologies, namely, Ultra-Wideband (UWB) and Wake-Up Radio (WUR), with a UAV that acts as a "ubiquitous local-host"of a Wireless Sensor Network (WSN). First, the UAV performs the localization of the sensor node via multiple UWB range measurements, and then it flies in its proximity to perform energy-efficient data acquisition. We propose an energy-efficient and accurate localization algorithm - based on multi-lateration - that is computationally inexpensive and robust to in-field noise. Aiming at minimizing the sensor node energy consumption, we also present a communication protocol that leverages WUR technology to minimize ON-time of the power-hungry UWB transceiver on the sensors. In-field experimental evaluation demonstrates that our approach achieves a sub-meter localization precision of the sensor nodes - i.e., down to 0.6 m - using only three range measurements, and runs in 4 ms on a low power microcontroller (ARM Cortex-M4F). Due to the presence of the WUR and the proposed lightweight algorithm, the entire localization-acquisition cycle requires only 31 mJ on the sensor node. The approach is suitable for several emerging Industrial Internet of Things application scenarios where a mobile vehicle needs to estimate the location of static objects without any precise knowledge of their position

Fully Onboard AI-Powered Human-Drone Pose Estimation on Ultralow-Power Autonomous Flying Nano-UAVs

Many emerging applications of nano-sized unmanned aerial vehicles (UAVs), with a few cm(2) form-factor, revolve around safely interacting with humans in complex scenarios, for example, monitoring their activities or looking after people needing care. Such sophisticated autonomous functionality must be achieved while dealing with severe constraints in payload, battery, and power budget (similar to 100 mW). In this work, we attack a complex task going from perception to control: to estimate and maintain the nano-UAV's relative 3-D pose with respect to a person while they freely move in the environment-a task that, to the best of our knowledge, has never previously been targeted with fully onboard computation on a nano-sized UAV. Our approach is centered around a novel vision-based deep neural network (DNN), called Frontnet, designed for deployment on top of a parallel ultra-low power (PULP) processor aboard a nano-UAV. We present a vertically integrated approach starting from the DNN model design, training, and dataset augmentation down to 8-bit quantization and deployment in-field. PULP-Frontnet can operate in real-time (up to 135 frame/s), consuming less than 87 mW for processing at peak throughput and down to 0.43 mJ/frame in the most energy-efficient operating point. Field experiments demonstrate a closed-loop top-notch autonomous navigation capability, with a tiny 27-g Crazyflie 2.1 nano-UAV. Compared against an ideal sensing setup, onboard pose inference yields excellent drone behavior in terms of median absolute errors, such as positional (onboard: 41 cm, ideal: 26 cm) and angular (onboard: 3.7 degrees, ideal: 4.1 degrees). We publicly release videos and the source code of our work

Fully Onboard AI-powered Human-Drone Pose Estimation on Ultra-low Power Autonomous Flying Nano-UAVs

Many emerging applications of nano-sized unmanned aerial vehicles (UAVs), with a few form-factor, revolve around safely interacting with humans in complex scenarios, for example, monitoring their activities or looking after people needing care. Such sophisticated autonomous functionality must be achieved while dealing with severe constraints in payload, battery, and power budget ( 100). In this work, we attack a complex task going from perception to control: to estimate and maintain the nano-UAV’s relative 3D pose with respect to a person while they freely move in the environment – a task that, to the best of our knowledge, has never previously been targeted with fully onboard computation on a nano-sized UAV. Our approach is centered around a novel vision-based deep neural network (DNN), called PULP-Frontnet, designed for deployment on top of a parallel ultra-low-power (PULP) processor aboard a nano-UAV. We present a vertically integrated approach starting from the DNN model design, training, and dataset augmentation down to 8-bit quantization and deployment in-field. PULP-Frontnet can operate in real-time (up to 135frame/), consuming less than 87 for processing at peak throughput and down to 0.43/frame in the most energy-efficient operating point. Field experiments demonstrate a closed-loop top-notch autonomous navigation capability, with a tiny 27-grams Crazyflie 2.1 nano-UAV. Compared against an ideal sensing setup, onboard pose inference yields excellent drone behavior in terms of median absolute errors, such as positional (onboard: 41, ideal: 26) and angular (onboard: 3.7, ideal: 4.1). We publicly release videos and the source code of our work

Sound Event Detection with Binary Neural Networks on Tightly Power-Constrained IoT Devices

Sound event detection (SED) is a hot topic in consumer and smart city applications. Existing approaches based on Deep Neural Networks are very effective, but highly demanding in terms of memory, power, and throughput when targeting ultra-low power always-on devices. Latency, availability, cost, and privacy requirements are pushing recent IoT systems to process the data on the node, close to the sensor, with a very limited energy supply, and tight constraints on the memory size and processing capabilities precluding to run state-of-the-art DNNs. In this paper, we explore the combination of extreme quantization to a small-footprint binary neural network (BNN) with the highly energy-efficient, RISC-V-based (8+1)-core GAP8 microcontroller. Starting from an existing CNN for SED whose footprint (815 kB) exceeds the 512 kB of memory available on our platform, we retrain the network using binary filters and activations to match these memory constraints. (Fully) binary neural networks come with a natural drop in accuracy of 12-18% on the challenging ImageNet object recognition challenge compared to their equivalent full-precision baselines. This BNN reaches a 77.9% accuracy, just 7% lower than the full-precision version, with 58 kB (7.2 times less) for the weights and 262 kB (2.4 times less) memory in total. With our BNN implementation, we reach a peak throughput of 4.6 GMAC/s and 1.5 GMAC/s over the full network, including preprocessing with Mel bins, which corresponds to an efficiency of 67.1 GMAC/s/W and 31.3 GMAC/s/W, respectively. Compared to the performance of an ARM Cortex-M4 implementation, our system has a 10.3 times faster execution time and a 51.1 times higher energy-efficiency.Comment: 6 pages conferenc

Automated Tuning of End-to-end Neural Flight Controllers for Autonomous Nano-drones

Convolutional neural networks (CNNs) are fueling the advancement of autonomous palm-sized drones, i.e., nano-drones, despite their limited power envelope and onboard processing capabilities. Computationally lighter than traditional geometrical approaches, CNNs are the ideal candidates to predict end-to-end signals directly from the sensor inputs to feed to the onboard flight controller. However, these sophisticated CNNs require significant complexity reduction and fine-grained tuning to be successfully deployed aboard a flying nano-drone. To date, these optimizations are mostly hand-crafted and require error-prone, labor-intensive iterative development flows. This work discusses methodologies and software tools to streamline and automate all the deployment stages on a low-power commercial multicore System-on-Chip. We investigate both an industrial closed-source and an academic open-source tool-set with a field-proofed state-of-the-art CNN for autonomous driving. Our results show a 2 reduction of the memory footprint and a speedup of 1.6 in the inference time, compared to the original hand-crafted CNN, with the same prediction accuracy

Enabling the heterogeneous accelerator model on ultra-low power microcontroller platforms

The stringent power constraints of complex microcontroller based devices (e.g. smart sensors for the IoT) represent an obstacle to the introduction of sophisticated functionality. Programmable accelerators would be extremely beneficial to provide the flexibility and energy efficiency required by fast-evolving IoT applications; however, the integration complexity and sub-10mW power budgets have been considered insurmountable obstacles so far. In this paper we demonstrate the feasibility of coupling a low power microcontroller unit (MCU) with a heterogeneous programmable accelerator for speeding-up computation-intensive algorithms at an ultra-low power (ULP) sub-10mW budget. Specifically, we develop a heterogeneous architecture coupling a Cortex-M series MCU with PULP, a programmable accelerator for ULP parallel computing. Complex functionality is enabled by the support for offloading parallel computational kernels from the MCU to the accelerator using the OpenMP programming model. We prototype this platform using a STM Nucleo board and a PULP FPGA emulator. We show that our methodology can deliver up to 60 7 gains in performance and energy efficiency on a diverse set of applications, opening the way for a new class of ULP heterogeneous architectures

An Energy-efficient Localization System for Imprecisely Positioned Sensor Nodes with Flying UAVs

This work investigates the capability of unmanned aerial vehicles (UAVs) to find and communicate with wireless sensor nodes positioned at unknown locations. In this scenario, the UAV acts as a mobile gateway that estimates the sensor node position using multiple ultra-wideband (UWB) range measurements, before flying in its vicinity to perform energy-efficient data acquisition. In addition to UWB, we use wake-up radio (WUR) to improve the sensor node's energy efficiency, keeping it in the always-on 'low-Activity' state when the drone is not nearby. The paper proposes a localization algorithm that consists of an iterative, noise-robust and computationally lightweight approach based on multi-lateration. Experimental evaluations performed on synthetic data demonstrate that our approach achieves a submeter localization accuracy using only three range measurements. We confirm this with an extensive in-field evaluation. The multilateration algorithm runs in 4 ms, in low power microcontrollers such as the ARM Cortex-M4F. The WUR and our energy-efficient algorithm enable the sensor node to consume only 31 mJ during the whole localization-Acquisition process. Our solution can be introduced in many other industrial applications where a mobile robot needs to estimate the location of imprecisely positioned objects

Fünfiiber-Drone: A Modular Open-Platform 18-grams Autonomous Nano-Drone

Miniaturizing an autonomous robot is a challenging task - not only the mechanical but also the electrical components have to operate within limited space, payload, and power. Furthermore, the algorithms for autonomous navigation, such as state-of-the-art (SoA) visual navigation deep neural networks (DNNs), are becoming increasingly complex, striving for more flexibility and agility. In this work, we present a sensor-rich, modular, nano-sized Unmanned Aerial Vehicle (UAV), almost as small as a five Swiss Franc coin - called Fünfliber - with a total weight of 18g and 7.2cm in diameter. We conceived our UAV as an open-source hardware robotic platform, controlled by a parallel ultra-low power (PULP) system-on-chip (SoC) with a wide set of onboard sensors, including three cameras (i.e., infrared, optical flow, and standard QVGA), multiple Time-of-Flight (ToF) sensors, a barometer, and an inertial measurement unit. Our system runs the tasks necessary for a flight controller (sensor acquisition, state estimation, and low-level control), requiring only 10% of the computational resources available aboard, consuming only 9mW - 13x less than an equivalent Cortex M4-based system. Pushing our system at its limit, we can use the remaining onboard computational power for sophisticated autonomous navigation workloads, as we showcase with an SoA DNN running at up to 18Hz, with a total electronics' power consumption of 271mW

An energy-efficient parallel algorithm for real-time near-optimal UAV path planning

We propose a shortest trajectory planning algorithm imple-mentation for Unmanned Aerial Vehicles (UAVs) on an em-bedded GPU. Our goal is the development of a fast, energy-efficient global planner for multi-rotor UAVs supporting hu-man operator during rescue missions. The work is based on OpenCL parallel non-deterministic version of the Dijkstra algorithm to solve the Single Source Shortest Path (SSSP). Our planner is suitable for real-Time path re-computation in dynamically varying environments of up to 200 m2. Results demonstrate the effcacy of the ap-proach, showing speedups of up to 74x, saving up to 98% of energy versus the sequential benchmark, while reaching near-optimal path selection, keeping the average path cost error smaller than 1.2%